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ABSTRACT 

Three research efforts that used computer-based 
simulations for maintenance training were in progress when this 
report was written: Game-Based Learning, which investigated the use 
of computer-based games to train electronics diagnostic skills;^Human 
Performance in Fault Diagnosis Tasks, which evaluated the use of ^ 
context-free tasks to train individuals to maintain actual equipment; 
and the Adaptive Computerized Training System^ which applied^ 
artificial intelligence techniques to electronic tTOublashootiDg__ 
training. These efforts had the common goal of teaching generali zable 

diagnostic skills rather than equipment-speci f ic' procedures . 

Preliminary findings suggest that each of the approaches can improve 
maintenance performance under certain conditions. Playing a logical 
game is an effective substitute for training in reading logic^ ci rcui t 
diagrams, and practice in solving context-free diagnostic tasks 
enhances subsequent performance when diagnosing faults ^" 
equipment^specific simulations, it was also found that feasibility of 
the Adaptive Computerized Training System had been demonstrated, . 
though the svstem had not yet received rigorous experimental 
evaluation, A 7-item reference list, 17 figures, arid 1 table are 
included. (Author/LMM) 
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FOREWORD 



The Manpower ancl Eaucational Systems Tt-chnical Area of the U.S. Arffiy Re- 
search Institute for the behavioral and Social Sciences (ARI) performs re- 

search ana developnient that includes the application of educational technology 
and simulations to military trainina. Resu^arch on the use Of compater-based 
simulations for maintenance tiainina i§ of special interest because the de- 
velopment and implementation of iuch simulations is seen as a means of reduc 
ing training time and costs. Cor.puter-based simulations can providegreater 
individualization of training than standard ap; roaches and can reduce the 
need for operational equipment during training; 

This report summarizes three on-going research efforts concerned with _ 
c<5mputer-based simulations for maintenance training. It is an edited version 
of an unpublished paper presented at the xUitary Operations Research Society 
Symposium in December 197 9. 

The first effort, Game-Erased Learning, is being conducted by in-house 
personnel. It is responsive to the requirem:enLs of RDTf.E ProDect 2S262717A790 
"Human Performance Effectiveness and Simulation V as described in the ARI FY 80 
Personnel Performance and Training Frogram: Basic Research and Exploratory 
Development . 

The second effort. Human Performance in Fault Diagnosis Tasks , is being 
conducted bv the University of Illinois as part of the ARI Research Themes 
Program. The research is responsive to_ th.e reguirements of RDT&E Project 
2ilGlte2B74F, "Basic Research ir: the P^ehavioral arid Social Sciences." 

In order to accom.rlish the third effort. Adaptive Computerized Training 
Svstem, ARI's resources were aua~cnted by contract with _Perceptronics , Inc . , 
an'organization selected as having unique capabilities for research and de- 
velooment in this area: The research is resf-onsi.e to the reqairements of 
FuTi^E Project 202637 4 4A795 , "Tr ai r. i ng S imu la t ion , " as described in the ABI 
FV BO Personnel Performance and Training Program: Advanced Development. 
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Reaui remeiit : 



Current Army maintenance tr.Aii.w is l^rqety e^niii'ment specific. The 
student is first tau^iht the s t e:. -by-s tei rrccedures necessary to locate the 
malfunction in a s].eciric item of oquij-ment, then si.endi^ time practicing, and 
finaJlv is tested on the equipment itself. Lr-cause the training content cor:- 
sists of eauipment-st'ecific i^rocedures rather tiian mere general troubleshoot- 
ing; lo^ucTthere ±s little iikvlihood of tran^f-r of the skills acquired to 
other itoms of equipnent that tiie student will encc^ur.ter on the job. This 
lack of tranf^fer is reflected m the relatively high proportion of functional 
com;.onents submitted for repair (-12^. in one study) In iddition, this cur- 
rent traininq approach makes inefficient use of student time, instructor time, 
and equipment, therobv inflatin.; costK ; Efficient methods for providing tram- 
inoj in.generalizable diagno-tic skills couid redu^e costs and im].rove on-the- 
30L skills. 



Proceavires: : 

Tiiree research efforts that Use 
nance traiiiing are being conducted, 
of computer-based games to teach dia> 
Fault Diagnosis Tasks evaluates the i 
viduals to maintain actual equiim^ent, 
tern an lies ^'artificial intelligenct ' 
t rai r.ing . 



r'jrputei-based simulations for mainte- 
;ame-Hased Learriing investigates the use 
iC'Stic skills. Hum:an F'erformance ii\ 
;e of context-free tasks to trairi indi-_ 

Tiie Adaptive Computerized Training Sys- 
teci^ni ?ues to electronic troubleshooting 



Findi ngs : 

The findings to date, while incomplete, su::gest that each of ^ the ai - 
preaches can improve maintenance performance under certair. conditions. Play- 
ing a lOQical game is an effective sutstitute_f or training m reading log: c 
circuit diagrams and practice solving context-free diagnostic tasks ennanc-3s 
subsequent performance when di j -ncs ing . f aul t s in equj ri^ent-specific simula- 
tions; Although the Adaptive Computerized Training System has yet to receiv. 
rigorous -xr:.er imental evaluation, its feasibility has been demonstrated. 



Utilization of Findings; 



Research will continue co in: rcve the descriVed techniques and_ to ^ evalu- 
ate them under conditions that are more representative of the Army / : aininq 
er-ircnment. Guidelines for deve lop 1 n p and using games for training will be 
develoced. The effects of training .with context-free tasks on the suh) sequent 
maintenance of actual equipment will be mves tigated , The cost anc training 
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effectiveness of the Adaptive Con, ^uterx zeci Tr.ininq System will be evaluated 
in aii eti-goin^.j course of instruction at an Army school. 
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^i-.-- i-e'-<--t G-^-i-ib.-s t!:r.v e:-f:. i-t.- ir. the aic-a of coni- a ter-based simu- 
l.ticnrfnr n.axnte:..nce tr..:.i^. currently belna conducted by the U-.S. An.y 
Kese->rch Irstitut. f c r the bc-i;«vicral and Social Sciences (ARI). ARI is in- 
te-e^-ted ir. the use cf computer-based sxtralacic.s for maintenance training_ 
because it .rovidcs a ...t.:.tial ne^.HB of overccr^inu sone of the problems in 
this area th.it t::e .-.rr'.y currently f scc-s . 

The first irofcler. for efficient :T..inter.ance training is the increasing 
r —' er an- con-;ievirv l C .-^rr/.- svste^.s. Not only is equipment becoming more 
^r^icult to maintain, hut there is more of it. The second problem . is that 
the skill ievel of the rod of manpower available for military service is de- 
.-■p-ninq- L-nless social or economic changes increase the desi rabili ty . of mili- 
ta-v service or the civilian educational system improves, future military 
m,alntenante rcrsonnel will re.ruire additiond or improved training to reach 
the same level of proficiency that current personnel achieve with existing 
trainin.;. The third problem iz that the number of People available for md 
tarv service is decreasinc and will continue, to do so untii the late 1990s _ 
(Baker, 19o0): ntl other factors being equal, there fore , the skil 1 levels of 
entering personnel are expected to be even lower in the future. Army schools 
must deal with these problems while facing increased operating costs and re- 
dU-Jtioi;B in j.ersonnel . 

The limited data available snuuest th.t improved r^aintenance performance 
could produce a substantial improvement in readiness. Dressel and Shields 
,1.7 0 measured organizational-level maintenance performance in a brigade- 
sde unit during a 1-year period. They collected data on selected end items 
of tl.e M551 armored reconnaissance airborne assault vehicle turneo in for^ex- 
chanco at a direct support mar n tenance . f acd i ty . -^^^^^^^ V ' f 

the total number of submissions and false removals (i tems submi.tea as defec 
tive that were, in fact, functioning properly). Overall, 42^ of the total 
item^ sur'mitted were false removals, and 32% of the re^^air time at_ tne direct 
su-;o-t level was soent determining that false removals were operating cor- 
Lctlv This represents an average of 1.5 hours per item, only .b hours less 
than the time required to repair a fadty item. Furthermore , 30% of equipment 
downtime was due 'to these false removals. Thus, a considerable amount of op- 
erational and maintenance time is lost because workinr equipment is removed 
for repair: 

'tvo caveats regarding these data shoaid be J^'^"^'^^^:^^^^ 
re-.'-eso^it a random samrlinq of firm:y maintenance perform.ance (adhcag_h there 
is^ nc' reason to believe that :hey are atypical) . Second,^ the _per rorm.ance^ 



observed undoubtedly reflects a nun^er of factors other than the qudit o 
maintenance training per se , such as command Pp-f-.^^""^^;™^^^^,^^^, 
personnel turbulence: The data do, however, clearly estaolisn the^.-.istence 

eyen to the militlry, but there are no ccm: arable d.ta available irom other 
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^igure^. Number of items submitted for re^a.r, by month (from Dressel . 
•hields , 1979) ; , 
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The Army currentlv follows primarily a hands-on and equipment-specific 
approach to maintenance training. The student is first taught the_ sequential 
steo-by-SLep procedures necessary to locate a mail function in a particular 
item of equipment, then spends time practicing, and fihaiiy is tested on the 
equipment itself . This apr roach has several advantages. It ensures, that the 
student has mastered certain prerequisite skills, such as the useof test 
eauipment. It teaches the student thephysical layout of the equipment and 
its relationship to functional or schematic diagrams. Finally, 'the student 
piractices assembling and disassembling an actual piece of equipment. 

There are also some disadvantages to this training approach , however . 
First, time constraints do not permit, the student to receive training on all 
of the equipment that is expected to be ericdun tered on the job. Because the 
training content consists of equipment-specific procedures, rather than___ 
troubleshooting logic, there is little likelihood of transfer of the skills 
accjuired to other items of equipment. Second, a s'obstantial amount of equip- 
ment, which could otherwise be used operationally, is required for training^ 
Third, instructors must spendiarge portions of their time inserting malfunc- 
tions into equipment, rather than actually conducting training . Fourth, a _ 
large amount of student time is spent assembling and disassembling the equip- 
ment, thus reducing the number of different malfunctions that they can ex- 
perience dori/ig training, 

" computer-based simulations provide a means to overcome these disadvan- 
tages. They can reduce the need for actual equicm.ent for training and can 
reduce the amount of instructor time devoted to "faulting" equipment, By_ 
eliminating the heed for time-consuming assembly and disassembly by thestu- 
dent, they provide more opportunity for the student to experiehceaiarger _ 
se^ of equipment faults. They can provide more efficient training by adapt- 
ing to individual differences in performance. More imrcrtant, they have the 
potential to provide the student with genera lizabie diagnosticskil Is that 
can be applied to a variety of items of equii:ment. Finally, because computer 
costs are rapidly decreasing they can be expected tc ^be inexpensive , in com- 
parison to the use of actual equipment, in the near future. 

. The first research effort to be described is called Game-Based Learning. 
Its Objective is to determine the training effectiveness ofgames and to de- 
velop procedures for the design and use of games for training (Baker, 1981). 
While the scope of this research goes beyond thearea of maintenance simula- 
tion, the early work has used maintenance training tasks. 

The second effort is Human Performance in Fault Diagnosis Tasks. One ^ 
objective of this research is to investigate the use of "context-free;- diag- 
nostic tasks to train individuals to maintain real equipment. ^This work is . 
being perfomed by Dr; William Rouse, of the University of Illinois, under a 
contract with AHI . 

The objective of the third effort, the Adaptive Comtuterized Training 
Sv-tem (ACTS), is to evaluate the use of artificial i nte 1 lic)enc. techniques 
for electronic troubleshooting training. Most c: ti-ie work has_been performed 
bv Dr. Amos Freedy , of Perceptronics , Inc., under contract to ARI . 
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Backqroujid 

The recent increase in the use of instructional gan,es within the edaca- 
tionai. industrial, and business communities can be attributed^to the h^gh 
ii'ri^t and motivation they appear 9---^^ .-'^^"^ , . 

pear to orovide a simulated environment within which -^'^'^ ° J^^^ 
information processing, and. decision-making sR.lls can f^/^; '^^^^"^ 
tained- Yet even though the use of gaming techniques for instruction ana 

: -nfng Ki'intrinsic Appeal , _there is ver,^ '^^f^y^^^l^^^'Z^, 
instructional games actually teach what they are designed 

aids to assist instructional game developers in designing training ettecti 
games do not exist because there is insufficient behavioral data on the critx 
cal learner and task variables in game-based learning. 

' The puruose of ARI research in game-based training is to conduct a sys- 
tematic assessment of the training effectiveness of games and to investigate 
tSt ful lotentiafas training media. To date, on. -experiment ^-^y" in- 
ducted (Simutis, Baker, Bersh S_ Alderman , 1979) and a second ^^^^^ 
Both have used a similar approach. measuring .^^^^ .J^:'^;^^-^^^.'""^^? first 
problem-solving game to sitnoiated electronics maintenance tasks 
experiment measured transfer of training to ^he reading of logic^circuit 

an <=nhnects receiveci some instruction and practice on this tasK oe 

controlled by providing all training, practice, and testing by computer. 
Experim ent 1 

in Experiment 1, 42 enlisted personnel from Fort_ Belvoir ,_Vft, were.ran^ 

Game, or Control Game. The e.-perimenTidi y .v-^^rrce sessions, followed by 

group participated in three i-hour training or practice sessions, 

a transfer task. 



- - — : f4^c^ r^nPTved 1 hour of instruction 

students --.^-^^l^^^^^^'^^lS^^V^i^^ (AI.D, OH, NAND , 
on the interpretation and meaning of five logi. ^ . diagrams. Following 
NDR, and INVERSION) and other features of logic "'^^"^ J , " ' 3 ■ ^ i^^els 

this, they received 2 hours of practice reacing logic ^ differed 

S complexity (A through _ F) r^^^^^^'^ o^di^'lrent S f^nc! 

in one or more of the following ways: the number ^^-^^-^-'^''^^^^.^-^-^^^^ - 
tions, the total number^ of _ logic elements , and tne number -^ ^J^ ^^^ 
.igure 3 shows a -ry simple l;^^^ ^^^^^^'^ inversions. A 

ent logic ^^^^^f ""f ^ c) "! Resented in Figure 4. It contains. 
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Session 



Group 


I 


II . 


III 


IV 


Logic Control 


logic syrTLbbl 
ins truction 


logic diagram 
practice 


logic diagram 
practice 


•transfer 
test 


Logic Game 


Masterrr.ind 


logic symbol 


logic diagram 


transfer 
test 




instruction 


practice 


Control Game 


Black j^ick 


logic symbol 
ins truction 


logic diagram 
practice 


trans fer 
test 



Eiqjure 2; Experimental design for Experiment 




Figure 3 . A very simple logic diagram I Level A) 
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in the Logic Controt gi-ouL- received practice, in all levels of diagrajns. Sub- 
jects in the other arcur.s practiced onlv oh Le^-el A, B, and C diagrams . Each 
diagram had U.e correct an.r.-erts) per.ciJed on the back. Subjects received 
feedback by checkinq their answers afttr com ieting tne diagrams. 




o- 

o- 



"5 lJ NANby- 



Figure 4. A moderately complex logic diagram (Level C) 




During the transfer test the subjects were given^ 1 hour to r.olve 40 new 
logic diagrams, ID diagrams from each of the four hignest ^"^^^ ^^^^^ 
ity. subjects began with the least comi-lex firoblems . No feedback regarding 
the correctness of their responses was given. 



subjects in the Logic Game group spent the first hour of their training 
plaving a computerized version of the game Mastennind , presented on^the^Uni- 
Lrsity of Illinois PLATO IV system. This game required tne sub3ects to de- 
termine a hidden sequence of four 'digits that had been. sampled wiChout re- 
Placement from the digits 1 through 8 and randomly assigned to positions 
^he first game trial ^as necessarily a guess, and after ^^^^__trxal feedback 
on the accuracy cf the guess was provided. The subjects were told (a) how 
°anv digits in the guess were correctly identified and P^^^^ ^ -^^J^,'^^^^^^,^ 
sec-uence oOsition and (b) how many digits in the guess were correctly^ identi 
fied but had been placed m the wrong P°-ticn: This inforrnat.on^^could be^ 
us^d to eliminate wrong alternatives on subsequent guesses- If the^subject 
had n^t ii^o^ered the'correct ..lution_wxthin eight tnals the^solatxon was 
shown and a new game begun. A different ranaom sequence o. digits was 



generated for each subject for each yame. Following the Mastermind session, 
the Logic Game group received the lo:?ic sv,'nbol_instructibh, the_logic diagram 
practice, and the transfer test. The only difference from thetraining re- 
ceived by the Logic Control group was that the practice included only the 
three least complex levels of circuit diagrams: 



The Control Gamegroup spent thefirst hour of training playing a com- 
puterized version of BX^^^k , also presented on the PLATO IV system. Sub- 
3ect§ played against the com! uter and were given an initial stake of $1,000. 
If they lost their stake, they were provided. with an additional $1 ,000 . The 
game rules were the traditional Flackiack rule§. The logic syrttooi_ training_ 
and practice provided to the control Gair,e group were identical to the train- 
ing and practice received by the" Logic Gane group. 



The transfer tas] was scored for both accuracy and performance. The ac- 
curacy score IS the proi.ortion of items attempted for which the subject gave 
the correct answer. The performancescore is the proportion of total items 
for which the subject gave the correct answer. 

Table 1 suirmarizes the croup means at each of the four levels of diagram 
conplexity (C, D, E, and F) used on the transfer task . The accuracy and per-^ 
formance scores are similar except at the highest level of complexity (Level F) 
Recall chat the subjects in the Logic Game and Control Game groups were not 
ex;:osed to diagrams as complex as those represented by Levels D,_E, and^F _ 
prior to the transfer task. Recall also that the level of complexity of the 
diagrams was confounded with time, with the most complexdiagrams _ being en-^ 
countered last. (The fact that some subjects were unable to complete all of 
the diagrams, then, explains the relatively large differences between the ac- 
curacy and perf crtriclnce scores found iii Level F.) 



Table 1 

Accuracy (A) and Per f onriahce- (P) Sc ores 



Grou- 



Diaqrarr. complexity 



Logic Control 
A 
P 



.92 
.89 



;9C 



;90 
.89 



.80 
.74 



.86 
.85 



Logic Game 
A 
P 



.61 
.56 



.71 
.70 



.B2 
.79 



;67 
:65 



.70 
.68 



Contrc I Game 
A 

P 



.65 
.63 



.62 



.6b 



.58 

."49 



.60 
.60 



IB 

o 

ERIC 



The pattern of significant results obtained indicates two f-^f j^'^Jf^' 
evident on the graph df the_ pe^ for:.ance scores , sh^wn in Fxgure First, 



Logic Coritrbl group was sape 



rior to the Contrbl Game gr^oup. The Logic Game 



qroup was in the middle, significantly worse _ than_ Ehe Logic Cont_rbl group xn 
t"ml of accuracy scores, but not perfbrmance scores, and sxgnxfxcantl^ bet^^ 
^«rthan the control Game qrouo in terns of performance scores , but not accu 
r.cy scores! S Second trend is that the Logxc Game group P-^ J^'^'^" , - 

S S "ontrol Game group_on th._dxagrams of i-^-^^^^^^.f "f^f J^^^ ^ y 
C and D)^ and more like the Logic Control group on dxaarams of high complexity 

(teveis E and . 
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PerforiT^ance scores as a fanccion of dia^^ram cdri: lexity 



'^hese results indicate that under certain condition, limited practice 

cannot be attributed to any ^general trano.er ^^^ects_ _ ^^g^if,. 
Because both game groups had the -^^-^'-'''-'-'^1°;^^^^,^.,^, it 
cantly different accuracy and P-f^-^^^^'^-^/^^.!-. !^;„f JirLccessful , 
also cannot be argued that limiteo practice wa. su..i.ie... ror 
performance on the transfer test. 

, - „r oc- frr^rii The Loejic Gaitic and Con- 

The results also -L-'^/^-f;^^°;,,;:r' ^he LoJc Control group was not. 
trol Game groups were exposea to a Logic'Game .roup do relatively 

was this -^-^^^^-''l .^'^l'^t^^^'J^ -^l.'i; i Ind F) than on the diagrams _ 
better on the high-complexity diagram. (Le ls^ anJ^ j ^ n,otivational 

of intermediate complexity ( Leve Is C ana D 7 .^^f ; .^l^i^.; "-^-e ^ the relatively 
letdown caused by transferring from a niqn.y cnaxxe..,,..o ...... - 
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boring task of reading logic circuit diagrams? A second experiment was de- 
signed to answer these questions. 



Experiment 2 

Figures 6 §how§ the design for Experiment 2, which is currently in pro- 
gress. It incorporates three major changes from Experiment 1. First, it in- 
cludes a "pure" Control group. This group will permit a determination of _ 
whether game plaving (either Blackjack or Mastermind ) is better than noaddi- 
tional practice. Second, all training and testing is being done by computer , 
eliminating the possibility that any effects could be caused by exposure to 
computers. The computer-based courseware is described in Yeager and Simutis 
(i97y)__. Finally, the transfer task has been changed from reading logic dia- 
grams ^to troubleshooting simulated logic circuits. 



Session 



Logic Control 


logic diagr'.'^m. 
instruction 6. 
practice 


f^".:lt i5^^iat:i>:-:': 
iv r ruct i or. S 
practice 


addi tional 
practice 


transfer 
task 


Logic Game 


logic diagram 
instruction £ 
practice 


Mastermind 


fault isolation 
instruction S 
T-ractice 


transfer 
task 


Control Game 


logic diagram 
instruction & 
prac tice 


Blacr: i^ck. 


fault isolation 
instruction & 
practice 


trans fer 
task 


Control 


logic diaigram 
instruction s 
prac tice 


f aa^ t isolation 
instruction & 
practice 


transfer task 





F igure &. Experimental design for Experiment 2; 



Future Directions 

The results of these experiments will provide data on_the ef f ectivenesi 
of a loaic=based game for training in electroni cs . tasks . Future research is 
planned:to replicate thefirst results and to explore the effects of individ- 
ual differences on game-based training. 

HUNIAI^J PERFOR:-':AiNCE IN FACTLT DIAGNOSIS TASKS 



Eack^rouna 

The research in "Haman Performance in Fault Diagnosis Tasks" assumes that ^ . 
some skills are common to all fault diaanosis tasks. Clearly, equipment-specific 
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sViil= are important; an electrorii^ technician would not be expected to be 
able to diagnose a faultv aircraft engine, nor would, ah aircraft mechanic be 
expected to be able to repair a radio, yet these tasks do have common elements. 

Dr William Bouse and his associates at the University of Illinois . have 
been evalaatuur the use or "contex.t^f ree" fault diagnosis tasks for. training 
in diagnostic skills. Context-free tasks do not represent any particular item 
or tvpe of equipment. Thi research strategy has been to examinethe^transfer 
of training from one context-free task to another, and from context-free tasks 
to equipment^specific tasks. In addition, the use of computer aiding is being 
investigated. Computer aiding represents a synthesis of unassisted human fault 
diagnosis and the use of automated test equipment. 



The Task ^ 

The term context-free task is best explained by example. Figure 7 slioWs 
a simple context-free task that will be referred to as Task 1. This display 
is presented to the student on a compoter terminal. The network consists of 
a"? X 7 matrix of components, nurtoered 1 through 49. The arcs. connecting the 
components are selected at random and are different each_time the _=tadent di- 
agnoses a new network. Each comronent is an AND gate. If allof the inputs 
to that component are 1, and if the component is good, theoatput of that_ 
component will be 1. If any of these conditions . is not satisfied ( that is, 
if any input is 0, or if the component is faulty), the output will be 0. 




The task of the student is to determine which component has failed. Stu- 
dents test arcs until the failed component is found; The final outputs of the 
network are show-n on ti^e riaht side of the display the upper left corner 
shows the actions raken by a student working on this iDroblem. The student 
first tested arc 22, 30 and obtained an output of 1. On the fourth test (arc 
31, 3B) a 0 (faulty) output wasfound: The next two tests allowed the student 
tb'determine that the failed component: was nunber 31. The student "entered the 
answer and was told that the response was correct. 



Wiien computer aiding is used with this task, the student is provided with 
an automated record-keepina system. An aiding algorithm asesthe structure of 
the network and the known outputs to eliminate those components that could not 
be faulty.. These components are "X'ed oat_" on the computer display . As the 
student makes further tests, additional ccmponents are eliminated^ To illus- 
trate how this could be done^ note that, even before any tests are made, cdm- 
pdhehts 45, 48, and 49 in Figure 7 are known to be good, because each of them 
has a 1 output. Any components having an input to those components must also 
be good, because all of the inputs to components 45, 48, and 49 must be 1 to 
get an output of 1. Thus, components 39,, 40, 41, and 42 can aiso be eliminated 
Additional components can be eliminated by working backward through the circuit 



In Task 1/ outputs are always fed forward to components of the _same type. 
Real system^ have different types of components , anc often have feedback loops. 
Task 2, presented in figure 8, is a more complex context-free task incorporat- 
ing two types of components and feedback loops. The rectangularcompbhents 
are AND gates, identical to the components o" the siniple task. The six-sided 
components are OR gates. _ OR gates produce an output of i if an£ of their in- 
puts is 1 and they have not failed. Teedback loops are disc pressent. For ex- 
ample, component 7 sends its output backward through the network to component 5 
The student's task is the same as in Task 1. 



The complexity of both of these tasks is easily varied by changing the 
numLer of components. 

In order to assess the transfer of training from these context-free tasks 
to euuipmer.t-specific tasks, a system to simulate "a_ variety of items; of real 
equipment was developeci. This system is called FAULT { Framework for Aidi ng 
the Understandina of Loqical Troubleshooting). FAUtT hastwo components. The 
first is a hard-copy schematic rf the equipment , such as the Turboprop Power- 
plant shown in Figure 9. The second component is an interactive display, 
shov;n in Figure 10. 

The student is initially given the general problem syTnptbms ; for example, 
the engine turns over but wil2 not start^ The student can then make_ inquiries 
about the functions of the system components; collect information about the 
functions of those components by. checking gauges, making continuity checks, 
and removing components for bench tests; and replace ccmponents. Each action 
has a cost based on the time that the action would require rnd, for replace- 
ments, the cost, of _ the replacement item. Students are instructed to repair 
the s^/stem at the lowest possible cost. 
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System: Turboprop 
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14 Tccn Generator $ 199 



F-iGmre 10, FAULT interactive disrtay (from Rouse et al . , 1980). 



The Research 

Five experiments have been con:. leted. Each examiried aspects of the_ 
context-free tasks and thei r use f or traininq. The -irst experiment used ' 
only the simple task (Task 1). Students solved problems having either 9, 25, 
or ^9 cbmpbhehts in a t rans f er-o f -t ra i n ing design in which half of the stu- 
dents first Used computer aiding and then transferred to the unaided task . 
The remaining students were first trained without computer aiding and then 
transferred to the aided task. The results indicated that the number of tests 
required to reach a correct solution increasingly deviated from the optimal 
solution as problem size increased. Computer aiding resulted in a lower nam-- 
ber of tests to solut ion duri ng the training trials, but not during the trans- 
fer trials. Computer aiding during training therefore enhanced subsequent 
unaided performance. 

In the second experiment the effects of requiring the students to oper- 
ate Under time constraints were examined. The time available to complete 
each problem was limited to either 30, 60, or 90 seconds. (It is relatively 
easy to solve these problems consistently in 90 seconds, but very difficult 
to do so in 30 seconds.) A clock was added to the display togive the stu- 
dents continuous knowledge oi the amount of time remaining. Task 1 was used^ 
Again, half the students, had computer aiding only during the first half of the 
experiment.. The others had computer aiiding only during the second half. All 
problems had 49 components. Students used more teststhan were necessary to 
solve the problems, and the effects of computer aiding did not transfer to the 
unaided condition. The interpretation of these results is that f orced-pacea 
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students employ l..._infornation, .nclua.n, .tudy of the computer aid; than 
do self-paced students. 

n^u f,-^--r fwo ex-erinents used engineering students as subjects. Experi- 

tiot; that transfer was exair.ined in only the ^^^^^^^ relative to the 

salts were .dent.cal to tho.. f ^^^^ J:'^ tn realS ' co;;,puter aiding 

perforTTiance on tl'^e unaided task. . 

The fourth experiment examined the ^^^-^^^ 
Tas. 1 Per^Or^nce_on_subse,uenyasK^^gr or^jn ^^Port^ ^^f^espects , 
semester aviation mecnanic trainees Bcrved^as^:^^ Performance de= 
Task 1 performance was 'Fd con^uter aiding during training im- 

clined -_P^-^^-. --^^^"^J,j;,^=J^-rtas n'o ransfer of training from the _ 
proved performance. _ »°^f ^ " . ^^""^ transfer had been shown by fourth- 

aided to the unaided Task Idxsp ays^ .o^sible interpretation of this finding 

semester students in ^^t^™" ^J-.^^J^ter Stodents^may have beeh using the 
is that the _less-experienced f irst semester^s y^^^^^ 

computer aiding only as a way to_rr..ke l^^^^^^^^' training was shown 
understand how _ the __aiding was he ping them^ ^f " me taken to reach the cor^ 
between Tasks 1 and 2 in tenns J^^t- „ade. Initially, the 

rect solution, but not in terms of ^'^^ ^ "^'"f^^f^^'f .lorSed worse on Task 2 
student._who had received computer aiding in Task 1 P-^°-^ ^^^^^ difference, 
than those who had received no Task 1 aiding, cat 
were reversed. The cSuse of this effect is not clear. 

Ev..eriment 5 exarnined the students' . "^"5 " " 'Siects 

^ped onlaiks 1 ahd_A - P.ULT e.uipment-specif ic^si^^ Sab3ects 

'-'- '''t'l^m^^'l^^Tl^^:^^^-^^^^ .nltransfer to three 
periment both Tasks i ana z ^^^p^.^d Students trained with computer 

Items of task w e Sbld to solve the ea.uipment--speci f ic 

aid.ng on- the ^°"^_^\^-^^f^^;f5han those students who hid not received com- 
problems at a high-cost procedures, such as bench 

puter aiding. These stuaent^ i-ki^ rn dauae readings more 

tests, and used the free in fo mat lon availab-e in gauge mg 

f recce nt ly . 

take place between the two levels o. ■ ^ sinalations. This 

from the context-free tasks to -'^^f''''^'^ ^ J^e problem at their 

transfer is most pronounced when ^^^^ ^f'^;^'^,;^ f^^^.s have diffi- 
own pace using computer aiding, It also a^, ^ ^.^.^^ assists them 

culty using correct (1) outputs efricier.a/. C-rv,u 

in rrraking use of this information. 



Future birec^iws 



e'^fects of computer 
Future research will -ntinue to inv. .i^.-^ ^^^^^l^^^^,,,, ".odels 
ng, atteSp£ to define problem corrx>l e.-.i . v , -nd d_.ei [ 
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of diaqnostic per f brrriancti : The maior thrust of this_research will be a cdn- 
tir;uatibn of the tran^^fcr studies to determine the degree of transfer from 
cbhtoxt-free tat-k:^ and equ i i-Tn- r: t -s^ eci f ic simulations to hahds-dn , actaai- 
equipment trouij Ilji^m. x -r uia , 

ADArrl'/K c:.Xf :."TE'r;IZi:D TP.i^^ININC SYSTEM 



Background 



The oreviousiy discussed research is concerned with .providing non- 
eaui;;ment-specific^ diagnostic training thrcuuh the use _of non-equipment- _ 
si'ecific diaqnostic tasks; The following research is directed toward pro- 
viding non-eqaipr^ent-srecif ic training using an equipment-specific task . 
Tills effort is called the Ada; tive Computerized Training System (ACTS). 

The ACTo also adds a new com: one n t to the training process — student and 
expert performance are modeled and th ese ■ student and expert models are used 
t(/ provide instructional feedback to the student and to direct the training 
process . 



This requires the use of artificial intelligence techniques. 



Artificial Intellioence (AI) techniques are algorithms (rules) that 

enable computers to exhibit "inte lligent"_ behavior . Exainples of intelligent 
behavior are understanding written English, playing chess, and learning 
(changing behavior as a result of experience): The AI techniques can pro- 
vide individualized instruction but do not require separate programing of 
the instructional logic for each^iesson: Extensive computer resources , how- 
ever, are required to sup; ort the u-e of AI techniques. In the past, this 
requirement has prevented the use of such systems outside a research 
e I. vi rcnmen t , 

The primarv objective of the ACTS research effort is to Improve the 
indi-iduaiization of maintenance training _ through the useofsfeme basic AI 
techniques that can be implemented on small-scale inexpensive computer 
s'" o cems , 



ACTC De sc r iptjon 



The student's task in ACTS training is to _troub leshoot an electronic _ 
circuit_bv making various test measurements, and rerlacing the malfunctioning 
-art The entire process is simulated by the ACTS. Neitherthe actual cir- 
cuit 'nor test equipment is required. The heart of the system is an adaptive 
conr-ter prSuram that models the Student's behavior, compares the moael of 
the" student to that of an exoert, and provides feedback to the stuaent to 



mar'.e 



is or her behavior more like that of the expert 



The ACTS is not being oroDosed as a complete troubleshooting training 
method. It will not train the student to use test equipment or to assemble 



or disassemble the equipment. It is designed to train the student m 
cision making during the troubleshooting process 



de- 



The ACTS consists of four major cdmponents r (a) the task model, (b) the 
rt model, (cj the student model, and (o) the instructional model. 
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Task abdil. TOe task Sod.l i, i simulation of ifie circuit on wHicB th, 
itua,rfe; trained,. The circuit =f "^"^f "'fi^" ^rL'" i 

the circuit. 
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Fiaure 11. Circuit diagram. 



i^rt Model. The second major component of the ACTS is a model of an 
Expert iviQQex . in ^ j .i^^^^^r^to the ex^:ert*s measurement choices 

expert troableshooter.^Th.s mode. ^^^^^^^^^ "^^^^^ ,,_nne observa- 
while troubleshooting the circuity It is deveiopeo^ni ^ 
tion by the con,puter of the expert's troubleshootin, beh.vxor. 

^Tli^T^^^s £ itua:;f Solves troubleshootin, problems on 
the ACTS. 

Wstructional Model. The instructional -^-^^^ -^^^^'^^ ^^^^ 
stuaeS^SdelT-^eternunes discrepancies between ^^^'^^i^^^' 
and modifies the instroictidnal sequences ror the s.uaent m order 
these discrepancies . 



i -^^^ -or '^rp'-^ision of description. 

the Army Research Institute. 



Modeling Behavior ; 

The uniaueness of the ACTS lies in the use of the student and. expert 
models. While the student ana expert models serve different functions and 
use different dat5, their Operation is identical. The original version of . 
the ACTS used anExpected Utility (Ed) approach to create the student and 
expert models. The current version uses a Multi-Attribute. Utility (MAU) 
approach. Both approaches "observe" human behavior and, through the aseof 
adaptive algorithms, derive sets of numbers that permit reproduction of that 

behavior. In the EU model, these numbers represent the human -^srelative 

preference for the outcomes of the actions that can be taken during the di- 
agnostic process. An action can be either a measurement or a module replace- 
ment. The EU model, however, has two drawbacks. First, behavior is repre- 
sented by a large set of numbers— at least two for every action that , can be 
taken. Second, the feedback that can be provided on the basii of this model 
is limited to indicating too high a preference or too low. a preference for 
various actions. Thus, a now model for the ACTS was sought to reduce compu- 
tations and increase feedback precision. 

In'the MAU model, a much smaller set Of nunbers is used. Each repre- .. 
sents an "attribute," or a general characteristic corr-mon to all actions. _The 
three attributes currently being Used are Decrease in Uncertainty, Fault Iso- 
lation, and Cost. Decrease in Uncertainty is the proportion of possible 
faults that is expected to be eliminated by an action. Fault Isolation is 
the proportion Of possible faulty modules that is expected to be eliminated 
by ari action. This differs from Decrease, in .Uncertainty because most modules 
can be faulty in several ways. Cost is the dollar cost of an action, based 
on the time reooired to take the ac-^on and the cost of replacement parts^ 
The use of the f-lAU model makes it possible to provide students with "higher 
order" feedback based on their relative preference-s for action attributes. 
For exa^iple, a student. can be tola that too little emphasis is being placed 
on the cost of the actions being considered. 

Developin g ACTS Tr aining 

Preparation of the ACTS for training require?, five steps, or tasks.. _ 
Fi-st a matrix showing the relationships between the possible faults and the 
resaiting measurement outcomes mUst be prepared. A sample is shown in iig- 
nre 12 An "L" indicates a lower than normal outcome, an."H" indicates a 
higher 'than normal outcome, and a blank indicates a normal outcome. 

The second task is to determine the probability of Occurrence of each _ 
possible fault. This can be done by examining maintenance records , consult- 
ing experts, or simply assuming that all faults are equally likely. 

The third task is to determine the cost ofeach measurement and module 
replacement. The^cost should include both the time required to take. an ac- 
tion and the cost of any replacement parts. 

Fourth, the computer must be programed to display the circuit diagram, 
and the fault matrix, fault probabilities, and action costs most be entered. 
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:-igure 12. Sample ACTS fault matrix (from Hopf-Weichel et al., 1980). 
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the fifth and final step is to train the expert model. This consists "of 
having a human expert solve a series of troubleshooting pro'blems on the .ACTS. 
During this step the expert mcdei learns the human expert *s preferences. When 
this is completed, the human expert is no longer needed. The expert model On 
the computer is ready to becin training the student. This process will be ex- 
amined from the viewpoint of the student. 



Training Sequences 

After receiving introductory trainingon the operation of the. circuit, 
the student is presented with a display similar to that shown in Figure 13. 
A simplified circuit diagram is shown at the top of the display. The lower 
left section contains a tsble used to present measurement results. The mea- 
surement points ( QUTPu t , Test Point J^, etc.) form the rows of this table. 
The types of measurement taken ( VGL tage, CURrent, and ^ES- istance) are shown 
in tiie columns. 
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Figure 13. Student display at the start of a problem (from Hopf-Weichel 
et al. , 1980) . 
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aUtbmticSUy prociaed to the student at the jtatt ot a P ^ ^ 

U then a.Ked t»_.tst ^'^^ ^^-.t^^Ti^r. 14. the 

may al§o be requested at this time. in une x ^ J_ shown the values of 

the three attributes for each ^^tion.. The circuit operating in 

TP9aCVR_ (a direct ^ current ,e.sure.et^at TP 9^wxth^^h 

a voltage regulation state) ^ is expeccea >u _^ ^ sg.OO. The 

faai-ts and 33% of the possible faulty n^ooules. It will cost 54 

- ;,q>:.».c^ TRA (replace the transformer) , _is expe(-ut;u 

fourth action_ considered, TRA (replace^ nossible faulty ffiodnies by 

nate 28% of the possible faults and 19% P"^^!,^';;^'^./ This action 

et^Mnating the possibility that the trans or^er could^ 

:-sta-e^Sa-re.St beSen ^^^d c | %^ 
the display. 
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LISTED BELOW ARE THE FOUR CONSIDERED ACTIONS AND THEIR 
ATTRIBUTE LEVELS. 
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•^10. do 

••98 . 00 



'RETURN. 



Figure 14. Student display after the student has selected an action 
(from Hopf-Weiche:t et al . , 1980). 



If the student requests help, the .expert provides a list of the four 
best actions to take. This is followed by more detailed information about 
those actions, as shov;n in Figure 15. 
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Figure 15. Student display during the Help sequence (from Hopf- 
Weichel et al. , 1980) . 



The student continues to take test measurements until the faulty module 
has been identified, then replaces the faulty module. At the completion of 
the problem, the student receives two types of feedback. First, the Ftudenfs 
cost to repair the circuit is compared withthat of the expert (see Figure 16) , 
Serond, as ishown in Figure 17, feedback based on a comparison of the student 
and expert models is provided. In this example, the student overemphasized 
Cncertainty Reduction and Fault Isolation, and unceremphasized Cost. The stu- 
dent wbuid then begin _another problem, with the process continuing until the 
student and expert models were sufficiently simi.li^r. 



Eva luatibft 

The key aaestion regarding the ACTS is whether it is an^effective train- 
ing system. s'ince the MAU model has just been impleinented , the only data _ 
available to date have been obtained using the EC model. Three ma^or findings 
were obtained. First, the "leafriirig" algorithns in the ACTS do learn to _pre- 
dict human performance. If performance is perfectly consistent, the_predic- 
tion of the learning algorithm will be perfect. Second, student performance 
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Figure 



'16. Cost-based feedback (from Hopf-Weichel et al., 1980) 
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improves with practice on the system, even when no feedback based on the stu- 
dent model is provided.. Third, simulation studies have shown that sirriilar 
sets of utilities produce similar trpubleshootihg strategies , while dissimi- 
lar sets of utilities produce dissimilar troiobleshobtihg stratiegies: This is 
a necessary prerequisite for the use of differences bietweeri two sets of util- 
ities (student and expert) as a basis for instructional feedback. 
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Figure 17. Model-based feedback (from Hopf-Weichel et al . , 1980) 



Fut ure Directions 

Future research on the ACTS will initially establish thetraining effec- 
tiveness of the ACTS. Research will include theinvestigation of transfer of^ 
ACTS training to actual equipment, the eftectsof varying the problem presenta- 
tion . sequence oh thebasis of student performance, and the effects of providing 
varying mixes of ACTS and actual-equipment training. Once this initial re- 
search is complete, the cost andtraininc ef f ectivenessof the ACTS in an on- 
going course of instruction at an Army school will be determined. 



SUNIMARY 

The Army's need for improved me thbds of providing maintenance training 
is expected to become more acute during the next two decades. At the same 
time, advances in computer technology are expected to result in low'-cost 
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=o„pa«? systems that can o»ed for trainin,. "'H=u,n ==cfi 

,t---^.^„^ o5?;^;pntlv Each uses comt-ater technology to provide in _ 
ment malfunctions_e.ficiently._^acn displavs, and simulation of main- 

dividUalized instruction, realistic graphic displays, _ rontext-free 
^- , - r^-y. , M different training approach: games, context rree 

duce ways to provide studentswitn gene _ _ ^ repair on 

applied to the variety of items or equipment that they wxll have 

the job. 
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